Picking system

ABSTRACT

A picking system is provided, which is capable of picking up an object even when the object is not registered in advance. The picking system includes: a picking device holding the object; an RGB-D camera acquiring three-dimensional point cloud data of the object to be picked up by the picking device; and a control device controlling the picking device based on a detection result by the RGB-D camera. The control device generates a geometric model of the object by combining simple geometric primitives while referring to the three-dimensional point cloud data, and calculates a holding position of the object for the picking device based on the geometric model.

TECHNICAL FIELD

The present invention relates to a picking system.

BACKGROUND ART

Conventionally, a picking system is known, which includes a pickingdevice that holds a workpiece (an object) and a control device thatcontrols the picking device (for example, see Patent Document 1).

The picking system in Patent Document 1 is configured to measure thethree-dimensional shape of a workpiece using a distance sensor andcompare a measurement result to a 3D CAD model of the workpiece, so thatthe position and the posture of the workpiece is recognized.

PRIOR ART DOCUMENT Patent Document

[Patent Document 1] JP 2010-069542 A

SUMMARY OF THE INVENTION Problem to Be Solved by the Invention

However, in the conventional picking system as described above, it isnecessary to register the 3D CAD model in advance so as to recognize theworkpiece. In this regard, there is still a room for improvement.

The present invention was made in consideration of the above problem, anobject of which is to provide a picking system capable of picking up anobject even when the object is not registered in advance.

Means for Solving the Problem

A picking system of the present invention includes: a picking deviceholding an object; a distance sensor acquiring three-dimensional pointcloud data of the object to be picked up by the picking device; and acontrol device controlling the picking device based on a detectionresult by the distance sensor. The control device generates a geometricmodel of the object by combining simple geometric primitives whilereferring to the three-dimensional point cloud data. Also, the controldevice calculates a holding position of the object for the pickingdevice based on the geometric model.

In this way, by generating the geometric model of the object andcalculating the holding position, it is possible to pick up the objecteven when the object is not registered in advance.

The above-described picking system may further include an image sensoracquiring image data of the object to be picked up by the pickingdevice. Geometric models of a plurality of types of objects andrespective holding parts of the geometric models may be registered inadvance in the control device. The control device may identify the typeof the object using the image data, and also may calculate the holdingposition of the object for the picking device taking into account acorresponding holding part of the registered geometric model of theidentified type of the object.

Effects of the Invention

With the picking system of the present invention, it is possible to pickup an object even when such an object is not registered in advance.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a schematic configuration of apicking system according to an embodiment.

FIG. 2 is a diagram explaining an example of a geometric modelregistered in a control device in the picking system of FIG. 1 .

FIG. 3 is a flowchart explaining operations to determine a holdingposition in the picking system of the embodiment.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, an embodiment of the present invention will be described.

A configuration of a picking system 100 according to an embodiment ofthe present invention is described with reference to FIGS. 1 and 2 .

The picking system 100 is configured to pick up an object (not shown) soas to perform, for example, automatic sorting and/or automatic transportof the object. The picking system 100 is provided to pick up one object(object to be held) located in a predetermined region. As shown in FIG.1 , the picking system 100 includes: a picking device 1; RGB-D cameras 2a and 2 b; and a control device 3.

The picking device 1 is provided to hold the object located in thepredetermined region. For example, the picking device 1 includes a robotarm and a hand, which are not shown in the Drawings. The hand isprovided at a tip of the robot arm so as to hold the object. The robotarm can control the position and the posture of the hand by moving thehand.

The RGB-D cameras 2 a and 2 b each take an image of the object locatedin the predetermined region so as to acquire an RGB-D image. The RGB-Dimage includes an RGB image (color image) and a depth image, and thushas information on each pixel depth of the RGB image. Also, the RGB-Dcameras 2 a and 2 b can convert an RGB-D image into a three-dimensionalpoint cloud data. Here, the RGB image is an example of “image data” ofthe present invention. Furthermore, the RGB-D camera 2 a is an exampleof a “distance sensor” and/or an “image sensor” of the presentinvention, and the RGB-D camera 2 b is also an example of a “distancesensor” and/or an “image sensor” of the present invention.

The RGB-D cameras 2 a and 2 b each take an image of the object fromdifferent angles. For example, the RGB-D camera 2 a takes an image of anobject located in a predetermined region from one side while the RGB-Dcamera 2 b takes an image of the object from the other side. That is,the two RGB-D cameras 2 a and 2 b are provided in order to prevent theouter shape of the object located in the predetermined region fromentering the blind spot.

The control device 3 controls the picking device 1 based on imagingresults by the RGB-D cameras 2 a and 2 b. The control device 3 includesan arithmetic section 31, a storage section 32, and an input-outputsection 33. The arithmetic section 31 executes arithmetic processingbased on a program and the like stored in the storage section 32. In thestorage section 32, the program and the like to control operations ofthe picking device 1 are stored. The input-output section 33 isconnected to the picking device 1, the RGB-D cameras 2 a and 2 b, andthe like. A control signal to control the operations of the pickingdevice 1 is output from the input-output section 33, and also theimaging results by the RGB-D cameras 2 a and 2 b are input into theinput-output section 33.

Then, the control device 3 calculates a holding position of the objectfor the picking device 1 based on the imaging results by the RGB-Dcameras 2 a and 2 b. The calculation of the holding position allowsappropriate picking-up of the object. In the storage section 32 arestored: a program to calculate the holding position of the object forthe picking device 1; a DB (database) 32 a for the program; and alearned model (not shown) that will be described later.

In the DB 32 a are stored, in association with one another, an IDindicating the type of the object, a geometric model of the object, anda holding part of the geometric model. That is, in the DB 32 a, the typeID of the object, the geometric model and the holding part arerespectively set as columns (items), and a plurality of records arestored. The records are registered in the DB 32 a in advance by, forexample, a user. Also, the geometric model of the object schematicallyand three-dimensionally represents the outer shape of the object, whichis generated by combining multiple simple geometric primitives with eachother. The simple geometric primitives include, for example: a cube; asphere; a cylinder; and a cone, whose orientation and size are variable.

As a specific example, when the type of the object is a “hammer”, ageometric model Mh is generated using two cylindrical objects C1 and C2,and a holding part Gp of the geometric model Mh is designated, as shownin FIG. 2 . The user registers the type ID of the object, generates thegeometric model Mh, and designates the holding part Gp. The holding partGp is a part of the object that is appropriate to be held by the pickingdevice 1. As one example, the centroid position of the geometric modelMh may be set as the holding part Gp. As to a plurality of objects, therecords as exemplarily described above are registered in advance.

As shown in FIG. 1 , the information on the position and the posture ofthe RGB-D cameras 2 a and 2 b (i.e. external parameters) is stored inadvance in the control device 3. The control device 3 integratesthree-dimensional point cloud data obtained by the RGB-D camera 2 a withthe three-dimensional point cloud data obtained by the RGB-D camera 2 b.Then, the control device 3 refers to the integrated three-dimensionalpoint cloud data of the object and generates a geometric model of theobject by combining the simple geometric primitives. The simplegeometric primitives include, for example: a cube; a sphere; a cylinder;and a cone, whose orientation and size are variable. That is, thegeometric model, which is approximated by the three-dimensional pointcloud data, is generated by combining the simple geometric primitiveswhile fitting the simple geometric primitives to the three-dimensionalpoint cloud data. This geometric model schematically andthree-dimensionally represents the outer shape of the object, which isgenerated by the multiple simple geometric primitives.

The control device 3 also identifies the type of the object using RGBimages (i.e. two-dimensional image data) obtained by the RGB-D cameras 2a and 2 b. This identification of the type of the object is performedusing the learned model (publicly known) stored in the storage section32. Then, when the control device 3 successfully identifies the type ofthe object and furthermore when the identified type of the object isregistered in the DB 32 a, the control device 3 calculates the holdingposition of the object based on the generated geometric model takinginto account the holding part of the geometric model of the objectregistered in the DB 32 a. On the other hand, when the control device 3unsuccessfully identifies the type of the object or when the identifiedtype of the object is not registered in the DB 32 a, the control device3 calculates the holding position of the object based on the generatedgeometric model.

Also, the control device 3 controls the picking device 1 such that thepicking device 1 holds the object by the calculated holding position ofthe object. That is, after operations to identify the holding positionas described below are completed, the control device 3 causes thepicking device 1 to perform picking operations by the holding positionof the object, which is calculated by the operations to identify theholding position. In other words, the control device 3 performs theoperations to identify the holding position before the pickingoperations by the picking device 1 is started. Thus, the holdingposition of the object at the time of picking operations is adjusted.

Operations to Identify Holding Position in Picking System

Here, a description will be given on the operations to identify theholding position in the picking system 100 according to this embodimentreferring to FIG. 3 . The operations to identify the holding positionare performed before the picking operations of the object located in thepredetermined region is started by the picking device 1. The respectivesteps described below are performed by the control device 3.

First, in step S1 shown in FIG. 3 , the imaging results by the RGB-Dcameras 2 a and 2 b are acquired. Specifically, the RGB-D cameras 2 aand 2 b take images of the object located in the predetermined region,and the imaging results are input into the input-output section 33. Theimaging result includes the RGB image and the three-dimensional pointcloud data. Then, the three-dimensional point cloud data from the RGB-Dcamera 2 a and the three-dimensional point cloud data from the RGB-Dcamera 2 b are integrated with each other.

Next in step S2, the geometric model of the object is generated bycombining the simple geometric primitives while referring to theintegrated three-dimensional point cloud data. That is, the geometricmodel, which is approximated by the three-dimensional point cloud data,is generated by combining the simple geometric primitives while fittingthe simple geometric primitives to the three-dimensional point clouddata. It is possible to improve the accuracy of the geometric model notonly by adding the simple geometric primitive, but also by removing thesimple geometric primitive. Thus, the geometric model is generated byadding or removing the simple geometric primitive(s) whose orientationand size are appropriately adjustable.

Next in step S3, the type of the object is identified using the RGBimages (two-dimensional image data) from the RGB-D cameras 2 a and 2 b.This identification of the type of the object is performed using thelearned model (publicly known). For example, when the RGB image is inputinto the learned model, the type of the object on the image is estimatedand also the certainty of the estimation result is calculated.

Next in step S4, it is determined whether the identification of the typeof the object using the RGB images has been successfully performed. Forexample, when the certainty of the estimated result calculated in stepS3 is not less than a predetermined threshold value, it is determinedthat the identification has been successfully performed. When it isdetermined that the identification has been successfully performed, theprocedure advances to step S5. On the other hand, when it is determinedthat the identification has been unsuccessfully performed, the procedureadvances to step S7.

Next in step S5, it is determined whether the identified type of theobject is registered in the DB 32 a. When it is determined that theidentified type of the object is registered in the DB 32 a, theprocedure advances to step S6. On the other hand, when it is determinedthat the identified type of the object is not registered in the DB 32 a,the procedure advances to step S7.

Next in step S6, the holding position of the object is calculated basedon the generated geometric model taking into account the holding part ofthe geometric model of the object registered in the DB 32 a. Forexample, the holding position of the object is calculated by comparingthe geometric model of the object that is generated in step S2 to thegeometric model of the object that is registered in the DB 32 a so as toapply the holding part of the registered geometric model of the objectto the generated geometric model of the object. In other words, theholding position of the object is calculated by applying the holdingpart designated by the user in advance according to the type of theobject to the generated geometric model. As a specific example, when thetype of the object is identified as a “hammer” using the RGB images, andfurthermore when the “hammer” is registered in the DB 32 a, the holdingposition of the object is calculated based on the geometric modelgenerated by referring to the three-dimensional point cloud data of theobject taking into account the holding part Gp of the registeredgeometric model Mh (see FIG. 2 ).

Also in step S7, the holding position of the object is calculated basedon the generated geometric model. For example, the centroid position inthe case where the density of the geometric model of the objectgenerated in step S2 is assumed to be uniform may be calculated as theholding position of the object. Also, the center position of the largestone of the simple geometric primitives constituting the geometric modelof the object generated in step S2 may be calculated as the holdingposition of the object.

Effects

In this embodiment as described above, the geometric model of the objectis generated by combining the simple geometric primitives whilereferring to the three-dimensional point cloud data. Also, the holdingposition of the object for the picking device 1 is calculated based onthe generated geometric model. Thus, it is possible to pick up theobject even when it is not registered in advance. Specifically, when theidentification of the type of the object has been unsuccessfullyperformed or when the identified type of the object is not registered inthe DB 32 a, it is also possible to appropriately pick up the object bycalculating the holding position based on the geometric model generatedby referring to the three-dimensional point cloud data.

Also in this embodiment, when the type of the object is registered inadvance, it is possible to improve the accuracy of picking-up bycalculating the holding position of the object taking into account theholding part of the registered geometric model of the object.

Also in this embodiment, it is possible to easily identify the type ofthe object by identifying the type of the object using the RGB images(two-dimensional image data).

Other Embodiments

The foregoing embodiment is to be considered in all respects asillustrative and not limiting. The scope of the invention is indicatedby the appended claims rather than by the foregoing description, and allmodifications and changes that come within the meaning and range ofequivalency of the claims are intended to be embraced therein.

For example, in the above-described embodiment, the two RGB-D cameras 2a and 2 b are provided. However, the number of the RGB-D cameras to beprovided is not particularly limited. For example, only one RGB-D cameramay be provided. In this case, the RGB-D camera may be attached to therobot arm. In this way, it is possible to capture the object frommultiple perspectives by taking images of the object by the RGB-D camerathat is being moved with the robot arm.

Also in the above-described embodiment, the identification of the typeof the object is performed using the RGB images (two-dimensional imagedata). However, the present invention is not limited thereto. Theidentification of the type of the object using the RGB images is notparticularly needed to be performed. In this case, after the geometricmodel of the object is generated by combining the simple geometricprimitives by referring to the three-dimensional point cloud data, theholding position of the object may be calculated based on the generatedgeometric model. That is, the steps S3 to S6 in the above-describedflowchart are not set, and the procedure may advance to step S7 afterthe step S2. Furthermore, since the RGB images are not required in thiscase, a distance sensor may be provided in place of the RGB-D camera inorder to acquire the three-dimensional point cloud data of the object.

Also in the above-described embodiment, the centroid position of thegeometric model Mh is set as the holding part Gp. However, the presentinvention is not limited thereto. The center position of the largest oneof the simple geometric primitives constituting the geometric model maybe set as the holding part. In this way, the holding part may be freelyset by the user.

Also in the above-described embodiment, the three-dimensional pointcloud data is input into the control device 3 from the RGB-D cameras 2 aand 2 b. However, the present invention is not limited thereto. Thecontrol device may calculate the three-dimensional point cloud databased on the RGB-D images that are input from an RGB-D camera.

Also in each of the RGB-D cameras 2 a and 2 b in the above-describedembodiment, an RGB image acquisition section that acquires RGB imagesand a depth image acquisition section that acquires depth images may beintegrally provided in one case, or may be separately provided inrespective cases.

Clauses

A method for identifying a holding position using a picking system, thepicking system including: a picking device holding an object; a distancesensor acquiring three-dimensional point cloud data of the object to bepicked up by the picking device; and a control device controlling thepicking device based on a detection result by the distance sensor,

-   -   the method comprising:    -   a step of acquiring, using the distance sensor, the        three-dimensional point cloud data of the object to be picked up        by the picking device;    -   a step of generating, by the control device, a geometric model        of the object by combining simple geometric primitives while        referring to the three-dimensional point cloud data; and    -   a step of calculating, by the control device, the holding        position of the object for the picking device based on the        geometric model.

The method for identifying a holding position as described above,wherein the picking system further includes an image sensor thatacquires image data of the object to be picked up by the picking device,and geometric models of a plurality of types of objects and respectiveholding parts of the geometric models are registered in advance in thecontrol device,

-   -   the method further comprising:    -   a step of acquiring, using the image sensor, the image data of        the object to be picked up by the picking device;    -   a step of identifying, by the control device, a type of the        object using the image data; and    -   a step of calculating, by the control device, the holding        position of the object for the picking device taking into        account a corresponding holding part of the registered geometric        model of the identified type of the object.

A picking method comprising the above-described method for identifying aholding position.

A program to cause a computer to execute the respective steps of theabove-described method for identifying a holding position.

Industrial Applicability

The present invention is applicable to a picking system including apicking device that holds an object and a control device that controlsthe picking device.

DESCRIPTION OF REFERENCE NUMERALS

1 Picking device

2 a RGB-D camera (distance sensor, image sensor)

2 b RGB-D camera (distance sensor, image sensor)

3 Control device

100 Picking system

What is claimed is:
 1. A picking system comprising: a picking deviceholding an object; a distance sensor acquiring three-dimensional pointcloud data of the object to be picked up by the picking device; and acontrol device controlling the picking device based on a detectionresult by the distance sensor, wherein the control device generates ageometric model of the object by combining simple geometric primitiveswhile referring to the three-dimensional point cloud data, and thecontrol device calculates a holding position of the object for thepicking device based on the geometric model.
 2. The picking systemaccording to claim 1, further comprising an image sensor acquiring imagedata of the object to be picked up by the picking device, whereingeometric models of a plurality of types of objects and respectiveholding parts of the geometric models are registered in advance in thecontrol device, the control device identifies a type of the object usingthe image data, and the control device calculates the holding positionof the object for the picking device taking into account a correspondingholding part of the registered geometric model of the identified type ofthe object.